Tracking device and tracking method

ABSTRACT

A tracking device comprising at least one memory and at least one processor which function as: an acquisition unit configured to acquire a viewed position, which is a position on a display viewed by a user; a tracking unit configured to track an object displayed on the display; and a control unit configured to perform control processing to control the tracking unit based on a degree of irregularity of a change of the viewed position.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a tracking device and a trackingmethod.

Description of the Related Art

Techniques, to detect a position at which a user is viewing (viewedposition) in a display unit of an electronic apparatus, are known. Forexample, there is a technique to track a specific subject in videoshooting in order to maintain focusing on the specific subject.

According to a technique disclosed in Japanese Patent ApplicationPublication No. H5-53043, in a case where there is a difference betweena position of a subject, which is being tracked in a video displayed ona display unit, and a viewed position by the user, the tracking targetis changed to another subject located closer to the viewed position.

In the technique disclosed in Japanese Patent Application PublicationNo. H5-53043, however, a subject that the user did not intend to focusmay be tracked if the user loses sight of the target subject duringshooting in the state where a plurality of subjects are moving in thevideo.

SUMMARY OF THE INVENTION

With the foregoing in view, it is an object of the present invention toprovide a tracking device that is capable of performing tracking controlin accordance with the intention of the user.

An aspect of the invention is: a tracking device comprising at least onememory and at least one processor which function as: an acquisition unitconfigured to acquire a viewed position, which is a position on adisplay viewed by a user; a tracking unit configured to track an objectdisplayed on the display; and a control unit configured to performcontrol processing to control the tracking unit based on a degree ofirregularity of a change of the viewed position.

An aspect of the invention is: a tracking method executed by a trackingdevice, which includes a tracking unit that tracks an object displayedon a display, the method comprising: an acquisition step of acquiring aviewed position on the display which is a position viewed by a user; anda control step of performing control processing to control the trackingunit based on a degree of irregularity of a change of the viewedposition.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a digital camera according to Embodiment 1;

FIG. 2 is a diagram depicting an example of a configuration of aline-of-sight acquisition unit according to Embodiment 1;

FIG. 3 is a diagram depicting an example of a configuration of aline-of-sight acquisition unit according to Embodiment 1;

FIG. 4 is an internal block diagram of an image processing unitaccording to Embodiment 1;

FIG. 5 is a flow chart of tracking processing according to Embodiment 1;

FIG. 6A is a diagram depicting a tracking frame according to Embodiment1;

FIG. 6B is a diagram depicting a detection frame according to Embodiment1;

FIG. 7 is a graph indicating a power spectrum according to Embodiment 1;

FIG. 8 is a graph indicating a viewed locus according to Embodiment 1;

FIG. 9 is a flow chart of execution determination processing accordingto Embodiment 2;

FIG. 10 is a diagram depicting a neutral network according to Embodiment3; and

FIG. 11 is a flow chart of execution determination processing accordingto Embodiment 4.

DESCRIPTION OF THE EMBODIMENTS

Embodiments will be described with reference to the accompanyingdrawings. The following embodiments, however, are not intended to limitthe present invention according to the Claims. Although a plurality offeatures are described in the embodiments, all of these features are notessential to the present invention. Further, these features may befreely combined. In the accompanying drawings, identical or similarcomposing elements are denoted with a same reference number, andredundant description thereof is omitted.

Embodiment 1

<Description of Configuration of Digital Camera> FIG. 1 is a blockdiagram depicting a configuration of a digital camera 100 (imagingapparatus) according to Embodiment 1. Instead of the digital camera 100,a tracking device (electronic apparatus) that can track a subject(object), such as a smartphone or PC (computer), that includes eachcomposing element of the digital camera 100, may be used.

In FIG. 1, a lens unit 150 is a lens unit equipped with a replaceableimage capturing lens. A lens 103 is normally constituted of a pluralityof lenses, but is indicated here as one lens. A communication terminal 6is a terminal for a lens unit 150 to communicate with the digital camera100 side. A communication terminal 10 is a terminal for the digitalcamera 100 to communicate with the lens unit 150 side. The lens unit 150communicates with a system control unit 50 via the communicationterminals 6 and 10. The lens unit 150 also controls an aperture 102using an internal lens system control circuit 4 via an aperture drivecircuit 2. Further, the lens unit 150 performs focusing by moving theposition of the lens 103 via an AF drive circuit 3. The lens systemcontrol circuit 4 stores information on the lens (e.g. information onfocal distance).

A shutter 101 is a focal plane shutter which can freely control theexposure time of an imaging unit 22 based on the control by the systemcontrol unit 50.

The imaging unit 22 captures images of a subject via the aperture 102and the lens 103. The imaging unit 22 is an image pickup elementconstituted of a CCD, a CMOS element, or the like, which converts anoptical image into electric signals. The imaging unit 22 includes pixelsgenerated by dividing a photoelectric conversion unit into a pluralityof regions, and each pixel corresponds to one micro lens. Thereby lightis divided and enters each pixel, and a phase difference detectionsignal can be acquired from the photoelectric conversion unit. Theimaging unit 22 can also acquire an imaging signal by adding signalsfrom each pixel. These pixels can play the dual roles of a focusdetection pixel and an imaging pixel. The imaging unit 22 may have onlythe imaging pixels, and in this case, the focus detection may beimplemented by a contrast method. In this way, the signals acquired bythe imaging unit 22 may be used not only for image capturing, but alsofor the exposure control and the focus detection control.

An A/D convertor 23 converts analog signals outputted from the imagingunit 22 into digital signals. The A/D convertor 23 outputs the converteddigital signals to an image processing unit 24 and a memory control unit15 as images (videos).

The image processing unit 24 performs resize processing (predeterminedpixel interpolation and demagnification processing) and color conversionprocessing on the image (data) outputted from the A/D convertor 23, oron the image outputted from the memory control unit 15. The imageprocessing unit 24 also performs predetermined arithmetic processingusing the captured images, and based on the result of this arithmeticprocessing, the system control unit 50 performs exposure control anddistance measurement control. Thereby through-the-lens (TTL) type autofocus (AF) processing, auto exposure (AE) processing and pre-flashemission (EF) processing are implemented. Furthermore, the imageprocessing unit 24 performs predetermined arithmetic processing usingthe captured images, and performs TTL type auto white balance (AWB)processing based on the result of the acquired arithmetic processing.

In this embodiment, the image processing unit 24 can perform thedetection processing and tracking processing for a subject based onimages (video; moving images). The internal configuration of the imageprocessing unit 24, to perform the detection processing and trackingprocessing for a subject, will be described later with reference to FIG.4.

The images (output data) outputted from the A/D convertor 23 are writtento a memory 32 via the image processing unit 24 and the memory controlunit 15, or via the memory control unit 15 alone. The memory 32 storesimages which were captured by the imaging unit 22 and converted intodigital signals by the A/D convertor 23, and also stores images to bedisplayed on the display unit 28. The memory 32 has a storage capacitythat is sufficient to store a predetermined number of still images, anda predetermined duration of moving images and sounds.

The memory 32 is also used as a memory for image display (video memory).A D/A convertor 19 converts the digital signals of images for imagedisplay, which are stored in the memory 32, into analog signals, andsupplies the analog signals to the display unit 28. In this way, theimages for display, written in the memory 32, are supplied via the D/Aconvertor 19, and are displayed by the display unit 28.

The display unit 28 performs display on such a display as an LCD inaccordance with the analog signals acquired from the D/A convertor 19.When the digital signals stored in the memory 32 are converted intoanalog signals by the D/A convertor 19 and sequentially transferred tothe display unit 28, the display unit 28 performs live view (LV)display. Hereafter images displayed in the live view display arereferred to as “live view images” (LV images). In the live view images,a subject, which the imaging unit 22 is currently capturing, isdisplayed.

The display unit 28 may be an electronic view finder to be looked intovia an eye piece (not illustrated), or may be a display disposed on arear face of the digital camera 100. The display unit 28 may includeboth the electronic view finder and the display on the rear face.

A non-volatile memory 56 is a memory which is electrically erasable andrecordable. For the non-volatile memory 56, an EEPROM or the like isused, for example. The non-volatile memory 56 stores constants, andprograms, and the like, for operating the system control unit 50. Forexample, the non-volatile memory 56 stores programs for executingvarious flow charts which will be described later in this embodiment.

The system control unit 50 controls each composing element of thedigital camera 100 by executing the programs stored in the non-volatilememory 56. A RAM is used for the system memory 52. The system controlunit 50 can develop the constants and variables for operating the systemcontrol unit 50, and the programs read from the non-volatile memory 56,in the system memory 52. The system control unit 50 also performsdisplay control by controlling the memory 32, the D/A convertor 19, thedisplay unit 28, and the like. A system timer 53 is a clock unit thatmeasures the time used for various controls and the time of the internalclock.

A power supply control unit 80 includes a battery detection circuit, aDC-DC convertor, a switch circuit to select blocks to be energized, andthe like. The power supply control unit 80 detects whether a battery(power supply unit 30) is installed, a type of battery, and the residualamount of battery power. The power supply control unit 80 controls theDC-DC convertor based on this detection result and the instructions fromthe system control unit 50, and supplies the power of the power supplyunit 30 to each composing element (including a recording medium 200). Apower supply unit 30 includes a primary battery (e.g. alkali battery,manganese battery, Li battery), a secondary battery (e.g. NiCd battery,NiMH battery, Li battery), and AC adaptor, and the like.

A recording medium interface (I/F) 18 is an interface with the recordingmedium 200 (e.g. memory card, hard disk). The recording medium 200 is arecording medium to record captured images, such as a memory card. Therecording medium 200 includes a semiconductor memory, a magnetic disk,or the like.

A communication unit 54 is connected with external devices via wirelesscommunication or cable communication, and transmits/receives videosignals and sound signals. The communication unit 54 is connectable witha network, such as an intranet and Internet. The communication unit 54can transmits images captured by the imaging unit 22 (including a liveview image) and images recorded in the recording medium 200. Thecommunication unit 54 can also receive images and various otherinformation from an external device.

An attitude detection unit 55 detects an attitude of the digital camera100 with respect to the direction of gravity. Based on the attitudedetected by the attitude detection unit 55, it can be determined whetheran image captured by the imaging unit 22 is an image that was capturedby the digital camera 100 held horizontally or an image that wascaptured by the digital camera 100 held vertically. The system controlunit 50 can attach the orientation information, which is in accordancewith the attitude detected by the attitude detection unit 55, to animage file of an image captured by the imaging unit 22, or can rotateand record the image. For the attitude detection unit 55, anacceleration sensor, a gyro sensor, or the like can be used. The motionof the digital camera 100 (e.g. pan, tilt, lift, still) can be detectedif the acceleration sensor or gyro sensor of the attitude detection unit55 is used. Further, the attitude detection unit 55 can detect arotation angle y around the z axis (yaw angle) of the digital camera 100in the xyz space where the direction of gravity is the z axis direction.The attitude detection unit 55 can also detect a rotation angle β in thevertical direction of the digital camera 100 (pitch angle around theyaxis along the lateral direction of the digital camera 100).Furthermore, the attitude detection unit 55 can detect the rotationangle a in the lateral inclination direction of the digital camera 100(roll angle around the x axis along the front/back direction of thedigital camera 100).

An operation unit 70 includes a plurality of operation members thatreceive operation instructions from the user. For the operation members,the operation unit 70 includes buttons (e.g. menu button, SET button)and a four-direction key, in order to execute menu selection, modeselection, reproduction of captured moving image, and the like. Forexample, if the menu button is pressed, a menu screen, in which varioussettings can be performed, is displayed on the display unit 28. The usercan intuitively perform various settings using the menu screen displayedon the display unit 28, the four-direction key and the SET buttons.

If a function icon is selected on the screen display on the display unit28, each operation member of the operation unit 70 can be operated as abutton (function button) to execute a function corresponding to thefunction icon. The function buttons are, for example, an end button, areturn button, an image switching button, a jump button, a previewbutton and an attribute change button.

Furthermore, for the operation members, the operation unit 70 alsoincludes a mode selection switch 60, a shutter button 61 and a powerswitch 72.

The mode selection switch 60 selects the operation mode of the systemcontrol unit 50 out of: a still image capturing mode, a moving imagecapturing mode, a reproduction mode, and the like. The modes included inthe still capturing mode are: auto image capturing mode, auto scenedetermination mode, manual mode, aperture priority mode (Av mode),shutter speed priority mode (Tv mode), and program AE mode (P mode).Various scene modes and custom modes, in which image capturing settingscan be performed for each image capturing scene, are also included. Theuser can directly select any one of these modes using the mode selectionswitch 60. The user may also select an image capturing mode list screenusing the mode selection switch 60 first, then select any one of theplurality of modes displayed on the list using another operation member.In the same manner, a plurality of modes may be included in the movingimage capturing mode.

The shutter button 61 includes a first shutter switch 62 and a secondshutter switch 64. If the first shutter switch 62 is turned ON inmid-operation of the shutter button 61, that is, in the half-depressedstate (image capturing preparation instruction), a first shutter switchsignal SW1 is generated. By the generation of the first shutter switchsignal SW1, the system control unit 50 starts the image capturingpreparation operation, such as an auto focus (AF) processing, autoexposure (AE) processing, auto white balance (AWB) processing, andpre-flash emission (EF) processing.

If the second shutter switch 64 is turned ON when the operation of theshutter button 61 completes, that is, in the fully-depressed state(image capturing instruction), a second shutter switch signal SW2 isgenerated. By the generation of the second shutter switch signal SW2,the system control unit 50 starts a series of operations of the imagecapturing processing, from reading signals from the imaging unit 22 towriting the captured image to the recording medium 200 as an image file.

The power switch 72 is an operation member to switch the power supply ofthe digital camera 100 between ON/OFF.

A line-of-sight acquisition unit 701 detects (acquires) a positionviewed by the user (viewed position; line-of-sight position) on thedisplay unit 28. FIG. 2 indicates an example of the line-of-sightacquisition unit 701. The line-of-sight acquisition unit 701 include animage sensor 701 a, a light-receiving lens 701 b, a dichroic mirror 701c, an eyepiece 701 d, an illumination light source 701 e, and a controlunit (not illustrated). In this state, a live view image captured viathe lens unit 150 is displayed on the display unit 28.

In order to detect the viewed position of the user who is viewing thedisplay unit 28 via the eyepiece 701 d, the illumination light source701 e projects infrared light to the eyeball 301 first. The infraredlight is reflected by the eyeball 301, and is further reflected by thedichroic mirror 701 c. Then this infrared light transmits through thelight-receiving lens 701 b, and enters the image sensor 701 a. Using theinfrared light that entered the image sensor 701 a, the image sensor 701a captures an image of the eyeball 301 and acquires an eyeball image.The control unit of the line-of-sight acquisition unit 701 extracts aregion of the pupil and the like from the captured eyeball image. Thecontrol unit detects a rotation angle of the optical axis of the eyeball301 of the user who looks into the finder visual field, and detects theline-of-sight of the user based on the detected rotation angle. Then thecontrol unit detects a position (region) on the display unit 28corresponding to the line-of-sight of the user (direction of the eye:direction the eye is viewing) as the viewed position. The line-of-sightacquisition unit 701 may capture an image of one eye of the user, orboth eyes of the user.

FIG. 3 indicates another example of the line-of-sight acquisition unit701 that is different from FIG. 2. Here the line-of-sight acquisitionunit 701 includes a camera 701 f, the illumination light source 701 e,and a control unit. In FIG. 3, a live view image captured by the lensunit 150 is displayed on the display unit 28. In FIG. 3, the camera 701f, which captures an image of a face 300 (eyeballs 301 and 302) of theuser observing the display unit 28, is provided on the rear face of thedigital camera 100. In FIG. 3, a range where the camera 701 f cancapture an image is indicated by a broken line. When light is emittedfrom the illumination light source 701 e to the face of the user, thecamera 701 f acquires an eyeball image (facial image). Then the controlunit of the line-of-sight acquisition unit 701 detects the line-of-sightand the viewed position of the user from the eyeball image.

In either case of the line-of-sight acquisition units 701 in FIG. 2 andFIG. 3, the eyeball image may be converted into digital signals by anA/D convertor (not illustrated) and be sent to the system control unit50. In this case, the system control unit 50 detects the viewed positionbased on the eyeball image. Therefore the configuration of theline-of-sight acquisition unit 701 is not limited to the configurationsillustrated in FIG. 2 and FIG. 3, as long as the information to detectthe line-of-sight (viewed position) of the user can be acquired.

(Image Processing Unit) A part of the internal configuration of theimage processing unit 24 to detect or track a subject, will be describedwith reference to the block diagram in FIG. 4. The image processing unit24 includes an image acquisition unit 410, a detection unit 411, and atracking unit 412.

The image acquisition unit 410 acquires video (live view image) from theA/D convertor 23. The image acquisition unit 410 may not only acquirevideo from the A/D convertor 23, but also acquire video from an externaldevice which is not the digital camera 100.

The detection unit 411 detects a subject (object) included in the videoacquired by the image acquisition unit 410. The subject detected by thedetection unit 411 is a subject (object) of a type which the user islikely to capture, such as a person, an animal and a vehicle. Thesubject detected by the detection unit 411 may be a subject of a typethat the user selected in advance. The detection unit 411 can detect asubject using such a conventional method as a convolutional neuralnetwork.

The tracking unit 412 tracks a specified subject in the video displayedon the display unit 28. The tracking unit 412 can also acquire theposition (x coordinate and y coordinate) of the subject that thetracking unit 412 is tracking in the video. The system control unit 50can switch between executing and stopping the tracking of the subject bythe tracking unit 412.

<Tracking Processing> Processing to control the tracking unit 412 totrack a subject intended by the user (tracking processing; trackingmethod; method for controlling tracking device), will be described withreference to a flow chart in FIG. 5. The processing of this flow chartis implemented by developing programs, which are recorded in thenon-volatile memory 56, in the system memory 52, and the system controlunit 50 executing the programs. Here it is assumed that the trackingunit 412 is tracking one subject in the video before the start of thisflow chart. At this time, a tracking frame that indicates the trackingsubject is displayed on the display unit 28.

In S501, the system control unit 50 controls the image acquisition unit410 and acquires video (live view image) from the A/D convertor 23. Theimage acquisition unit 410 may acquire the video from the memory 32 orthe like, instead of acquiring the video from the A/D convertor 23.

In S502, the system control unit 50 acquires the viewed position of theuser from the line-of-sight acquisition unit 701.

In S503, the system control unit 50 controls the tracking unit 412, andacquires a position (display position; position information) of thesubject (tracking subject) which the tracking unit 412 is tracking inthe video acquires by the image acquisition unit 410. In a case wherethe tracking unit 412 stops tracking the subject, the detection unit 411acquires the current position of the subject which the tracking unit 412was tracking at the point of stopping the tracking.

In S504, the system control unit 50 acquires a degree of irregularity ofthe locus of viewed position (line-of-sight) (viewed locus; change ofthe viewed position; change of line-of-sight). The method of acquiringthe degree of irregularity of the viewed locus will be described later.

In S505, the system control unit 50 determines whether the degree ofirregularity of the viewed locus is a predetermined threshold THr orless. In the case where the degree of irregularity of the viewed locusis the threshold THr or less, the viewed position is changing withregularity, hence the system control unit 50 determines that theline-of-sight of the user is tracking the subject. In the case where thedegree of irregularity of the viewed locus is more than the thresholdTHr, on the other hand, the viewed position is changing withirregularity, hence the system control unit 50 determines that the useris not tracking the subject (user has lost sight of the subject). If thedegree of irregularity of the viewed locus is the threshold THr or less,processing proceeds to S508. If the degree of irregularity of the viewedlocus is more than the threshold THr, processing proceeds to S506.

In S506, the system control unit 50 controls the tracking unit 412, andstops the tracking by the tracking unit 412. Then the system controlunit 50 stops the display of the tracking frame on the display unit 28.If the tracking by the tracking unit 412 has already stopped, the systemcontrol unit 50 executes no processing in S506.

In S507, the system control unit 50 controls the detection unit 411 anddetects one or a plurality of subjects in the video, and displays aframe (detection frame) which indicates each one of the subjects on thedisplay unit 28. The system control unit 50 need not necessarily displaythe detection frame, and may display each position of one or a pluralityof subjects so that the user can easily recognize the position(highlight display). Thereby the user can easily recognize the positionof each subject included in the video, and therefore the line-of-sightof the user can easily track the desired subject. At this time, thesystem control unit 50 may inform (notify) the user by sound or imagethat the viewed locus is irregular. When the processing in S507 ends,the processing in S501 to S505 is executed again. In other words, thesystem control unit 50 controls such that the tracking stop statecontinues and the detection frame indicating each subject iscontinuously displayed until the viewed locus changes with regularity.

In S508, the system control unit 50 determines whether a difference DFbetween the position of the subject acquired in S503 and the viewedposition acquired in S502 is a predetermined threshold THp or less. Inthe case where the difference DF is the threshold THp or less, thesystem control unit 50 determines that the line-of-sight of the user istracking the subject of which position was acquired in S503. In the casewhere the difference DF is more than the threshold THp, on the otherhand, the system control unit 50 determines that the line-of-sight ofthe user is tracking a subject that is not the subject of which positionwas acquired in S503. If the difference DF is the threshold THp or less,processing proceeds to S509. If the difference DF is more than thethreshold THp, processing proceeds to S510.

In S509, the system control unit 50 controls the tracking unit 412, andtracks the subject of which position was acquired in S503 (continuouslytracks the subject).

In S510, the system control unit 50 determines that the user is trackingthe subject closest to (or in the proximity of) the viewed position, andcontrols the tracking unit 412 so as to track the subject closest to theviewed position (subject in proximity) (changes the tracking subject).

In S511, the system control unit 50 displays, on the display unit 28, atracking frame to indicate the subject which the tracking unit 412 istracking.

Therefore in S509 and S510, the system control unit 50 determines thatthe subject corresponding to the viewed position is the main subject(subject that the user intends to track (image intended to capture)),and displays a tracking frame to indicate the subject corresponding tothe viewed position, on the display unit 28, in S511.

In S512, the system control unit 50 selects a focus detection regionclosest to the tracking frame, and acquires (detects) the focus state(defocusing amount and direction thereof) using a signal acquired by theimaging unit 22 (phase difference detection signal).

In S513, the system control unit 50 calculates the lens driving amountand lens driving direction corresponding to the defocusing amount anddefocusing direction acquired in S512. The system control unit 50controls the position of the lens 103 in accordance with the calculatedlens driving amount and lens driving direction, whereby the focaldistance is adjusted and the imaging unit 22 is controlled to executeimage capturing.

As described above, according to this embodiment, it is determinedwhether the line-of-sight of the user is tracking the subject or notbased on the irregularity of the viewed locus, and based on this result,the tracking by the tracking unit 412 is controlled. If the degree ofirregularity of the viewed locus is high (if it is determined that theline-of-sight of the user is not tracking the subject), the digitalcamera 100 stops tracking of the subject by the tracking unit 412, anddisplays the detection frame to indicate each subject included in thevideo. Therefore the possibility of tracking of the subject unintendedby the user is suppressed in the digital camera 100, and trackingcontrol in accordance with the intension of the user can be implemented.

<Display of Tracking Frame and Detection Frame> FIG. 6A and FIG. 6B arediagrams for describing display of the tracking frame and detectionframe on the display unit 28. Here it is assumed that the digital camera100 is capturing a sports scene. FIG. 6A and FIG. 6B indicate sceneswhere a person 601 and/or a person 603 are playing sports using a ball602. Detection frames 621 to 623 are display items that indicate thesubjects displayed on the video. A tracking frame 611 is a display itemto indicate a subject that the digital camera 100 is tracking.

In FIG. 6A, the viewed locus 600 is regular (YES in S505), and thedistance between the viewed position and the person 601 (subject) isshort (YES in S508). Therefore it is more likely that the subjectindicated by the tracking frame 611 and the subject that the userintends to capture (main subject) are both the person 601. In this case,in the display according to this embodiment, the tracking frame 611tracks the person 601 (S509, S511), therefore the user can easily trackthe person 601 (main subject) continuously by viewing the tracking frame611. At this time, the detection frame is not displayed so that the usercan focus on tracking the target.

In FIG. 6B, on the other hand, the viewed position moves randomly andthe degree of irregularity of the viewed locus 600 is high (NO in S505).In this case, it is more likely that the user has lost sight of thetarget subject (main subject) to be captured. Therefore the digitalcamera 100 stops display of the tracking frame 611, and displays thedetection frames 621 to 623 which indicate the subjects included in thevideo (S507) so that the user can easily find the target subject (mainsubject) to be captured (S506). Thereby the user can find the mainsubject from the subjects indicated by the detection frames 621 to 623,and can find the main subject more easily compared with the case of notdisplaying the detection frames.

The system control unit 50 may be an arbitrary display item (e.g. a starsymbol, circle) at the position of each subject, instead of thedetection frame and the tracking frame, since critical here is that theuser recognizes the position of each subject. The system control unit 50may change the color and/or thickness between the tracking frame anddetection frame displayed on the display unit 28. Further, the systemcontrol unit 50 may differentiate the display between the tracking frameand detection frame by another method. Thereby the user can more easilyrecognize which one: the detection frame and the tracking frame, isbeing displayed, and can more easily find the main subject.

<Viewed Locus Acquisition Processing> Processing (acquisition method;calculation method) to acquire the degree of irregularity of the viewedlocus (change of viewed position) performed by the system control unit50 will be described in detail.

(Case of Using Frequency to Acquire Degree of Irregularity) First a caseof using frequency information of the viewed locus in the time axisdirection, to acquire the degree of irregularity of the viewed locus,will be described. When time is t and the coordinates of the viewedposition at each time is (x(t), y(t)), the result X(w) of performingFourier transform on x(t) and the result Y(w) of performing Fouriertransform on y(t) are given by the following Expression (1) andExpression (2).

X(ω)=Σ_(k=0) ^(N−1) x(t _(k))exp (−2πi(t ₀ +kΔt)ω)   Expression 1)

Y(ω)=Σ_(k=0) ^(N−1) y(t _(k))exp (−2πi(t ₀ +kΔt)ω)   (Expression 2)

Here Δt is a time interval to acquire a viewed position, and frequency wis a frequency in a period from time to to time t₀+NΔt. As indicated inExpression 3, the power spectrum P(ω) is defined by a sum of a square ofthe absolute value of X(ω) and a square of the absolute value of Y(ω).

P(ω)=|X(ω)|² +|Y(ω)|²   (Expression 3)

In FIG. 7, the abscissa indicates the frequency w and the ordinateindicates the power spectrum P(ω). For the degree of irregularity, thesystem control unit 50 calculates the sum S of the power spectrum P(ω)that is not less than a predetermined frequency ω₀. Then in S505, thesystem control unit 50 determines whether the sum S (degree ofirregularity) is the threshold or less.

(Case of Using Auto Correction for Degree of Irregularity) An example ofusing a value based on the auto correlation of the viewed locus for thedegree of irregularity will be described next. FIG. 8 is a graphplotting x(t), which is an x position (x coordinate) of the viewed locusat time t with respect to the abscissa of time t. At time width 801 andtime width 802, which are adjacent to each other, the auto correlationbecomes close to 1 if the viewed locus is regular, and becomes closer to0 if the view locus is random.

The absolute value R(T) of the auto correction is given by Expression 4.Here Δt is a time interval to acquire the line-of-sight, and T=NΔt is awidth of time to calculate the correlation.

R(T)=|Σ_(k=0) ^(N−1) x(t ₀ +kΔt+T)x(t ₀ +kΔt)|+|Σ_(k=0) ^(N−1) y(t ₀kΔt+T)y(t ₀ +kΔt)|   (Expression 4)

In Expression 4, the auto correlation is determined for adjacent periodsof time width T, but may be periods close to each other, even if theperiods are not adjacent. The left side (x component) and the right side(y component) of Expression 4 may be added with weighting, instead ofsimply performing an addition. In S505, the system control unit 50determines an inverse number of the absolute value R(T) of the autocorrelation as the degree of irregularity of the viewed locus, anddetermines whether the degree of irregularity is the threshold or less,for example.

(Case of Using Other Information for Degree of Irregularity) For thedegree of irregularity, representative values (mean value, mode orcenter value) of the velocity vector (vx, vy)=(dx/dt, dy/dt) of theline-of-sight or the magnitude of the acceleration vector (dvx/dt,vy/dt) in the time width T may be used as the degree of irregularity.Here d/dt indicates differentiation with respect to time t. The degreeof irregularity of the viewed locus may be determined by arbitrarilycombining the line-of-sight vector, the acceleration vector, the autocorrelation, the frequency, and the like.

As described above, according to this embodiment, tracking in thedigital camera 100 is controlled based on the degree of irregularity ofthe viewed locus. Thereby tracking in the digital camera 100, inaccordance with the intention of the user, can be implemented.

Modification 1

To determine the degree of irregularity of the viewed locus, locusinformation on the position of the subject (display position) detectedby the detection unit 411 may be used instead of the information on theviewpoint locus. For example, the system control unit 50 calculates thecross-correlation between a vector of the viewed locus at a certain timewidth T=NΔt and a vector of the locus of the position of the subject.Then the system control unit 50 uses a value based on the absolute valueof the cross-correlation (e.g. inverse number of the absolute value)between each vector of the locus of all the positions of the subjectdetected by the detection unit 411 and the vector of the viewed locus.For example, an inverse number of the largest value of the absolutevalues of the cross-correlation between each of the vector of the locusof the positions of the subject detected by the detection unit 411 andthe vector of the viewed locus is regarded as the degree of irregularityof the viewed locus. In other words, in this modification, the viewedlocus is determined as a regular locus conforming to the locus of theposition of the subject, if the viewed position and the position of thesubject are changing in the same way. Here the cross-correlationR(t_(o)) in a period from time t_(o) to time t₀+NΔt is calculated by thefollowing Expression 5.

R(t ₀)=|Σ_(k=0) ^(N−1) x _(e)(t ₀ +kΔt)x _(o)(t ₀ +kΔt)|+|Σ_(k=0) ^(N−1)y _(e)(t ₀ +kΔt)y _(o)(t ₀ +kΔt)|   (Expression 5)

Here (x_(e)(t), y_(e)(t) indicates a position of the viewpoint at timet. (x₀(t), y₀(t)) indicates a position of the subject at time t. Thecross-correlation R(t₀) as well may be calculated by a weighted additionof the x component (left side) and the y component (right side).Further, cross-correlation between the motion vector (velocity vector oracceleration vector) of the viewed position and the motion vector of thedetected subject may be used.

Embodiment 2

A digital camera 100 according to Embodiment 2 changes whether theprocessing to determine execution of tracking described in Embodiment 1(tracking control processing in S504 to S507) is performed or not, inaccordance with the locus (change) of the position of the subjectincluded in the video. This digital camera 100 will be described next.Furthermore, this digital camera 100 according to Embodiment 2 changesthe threshold THr to determine irregularity of the viewed locus in S505,in accordance with the locus (change) of the subject included in thevideo.

For example, in a scene where a subject is moving with near regularity,it is easy for the user to continuously track this subject, andtherefore the line-of-sight of the user is more likely to be trackingthe subject indicated by the tracking frame. This means that it isprobably appropriate for the tracking unit 412 to continue the currenttracking, and the need to perform the tracking control processingaccording to Embodiment 1 is low.

Further, in a scene where a plurality of subjects move in variousdirections (e.g. sports scene), the degree of irregularity of the viewedlocus tends to become high, even if the line-of-sight of the user isappropriately tracking the subject. Therefore in such a case, even ifthe line-of-sight of the user is tracking the subject moving in variousdirections, it may be determined in error that the line-of-sight of theuser is not tracking the subject since the viewed position movesirregularly. In this case, it is better that the threshold THr, todetermine the irregularity of the viewed locus, is larger than in othercases.

<Execution Determination Processing> Processing to determine whether ornot the tracking control processing is executed (execution determinationprocessing) will be described with reference to the flow chart in FIG.9. Description on the processing steps the same as Embodiment 1 will beomitted. In the case where a plurality of subjects are included in animage, “the degree of irregularity of the subject locus” in thefollowing description should be interpreted as “the representative value(e.g. mean value, center value, mode) of the degree of irregularity ofeach locus of the positions of the plurality of subjects”.

In S901, the system control unit 50 acquires (calculates) the degree ofirregularity of the subject locus, and determines whether the degree ofirregularity of the subject locus is a first threshold THrl or more. Thefrequency of the subject locus, the auto correlation of the subjectlocus in the time direction, the velocity vector and the accelerationvector can be used to determine the degree of irregularity of thesubject locus, in the same manner as the viewed locus according toEmbodiment 1. In the case where the degree of irregularity of thesubject locus is the first threshold THr1 or more (the threshold ormore), processing proceeds to S902. And in the case where the degree ofirregularity of the subject locus is less than the first threshold THr1(less than the threshold), processing proceeds to S905.

In S902, the system control unit 50 determines whether the degree ofirregularity of the subject locus is less than the second thresholdTHr2. Here the second threshold THr2 is a value larger than the firstthreshold THr1. In the case where the degree of irregularity of thesubject locus is less than the second threshold THr2, processingproceeds to S903. In the case where the degree of irregularity of thesubject locus is the second threshold THr2 or more, processing proceedsto S904.

In S903, the system control unit 50 executes the processing in S504 toS513 including the tracking control processing (S504 to S507). After theprocessing in S507 is executed, processing proceeds to S501 in this flowchart.

In S904, the system control unit 50 increases the threshold THr for thedetermination in S505 to be more than the case where S902 is YES (thecase where the degree of irregularity of the subject locus is the firstthreshold THr1 or more, and less than the second threshold THr2). Inthis way, the system control unit 50 increases the threshold THr, whichis a threshold to determine the irregularity of the viewed locus, if thedegree of irregularity of the subject locus is high. Thereby in such ascene, where the subject moves irregularly, it becomes less likely to bedetermined in error that the line-of-sight of the user is not trackingthe subject (NO in S505), even if the line-of-sight of the user isactually tracking the subject.

In S905, the system control unit 50 does not execute the tracing controlprocessing (S504 to S507). This is because in the case where processingadvanced to S905, the locus of the position of the subject is quiteregular, and the line-of-sight of the user can easily track the subject,that is, it is unlikely that the user is not tracking the subject. Herethe system control unit 50 executes processing in S508 to S513, which isprocessing other than the tracking control processing (S504 to S507).

As described above, according to Embodiment 2, the digital camera 100changes whether or not the tracking control processing (S504 to S507) isexecuted, and changes the threshold THr to determine irregularity of theviewed locus, in accordance with the subject locus. Thereby theprocessing amount generated by unnecessary tracking control processingcan be eliminated, and it can be appropriately determined whether or notthe line-of-sight of the user is tracking the subject.

Embodiment 3

According to Embodiment 3, in the case where it is determined that thedegree of irregularity of the viewed locus is high in S505 in Embodiment1, the system control unit 50 determines the main subject (subject thatthe user intends to capture (track)), using a machine learning deviceincluded in the digital camera 100. Then the system control unit 50tracks the determined main subject, and displays the tracking frame. Thepresent invention is not limited to this, and the system control unit 50may determine the main subject using the machine learning device, evenif the viewed locus is regular (without determination in S505).

In the following, a case of using a neural network, which is a method ofmachine learning for the machine learning device, will be described, butsuch a regression procedure as linear (non-linear) regression may beused.

FIG. 10 is a diagram depicting an example of a structure of a neuralnetwork. The neural network illustrated in FIG. 10 includes an inputlayer 1001, an intermediate layer 1002, an output layer 1003 and neurons1004. Each connecting line 1005 indicates a connection relationshipbetween neurons 1004. Here only representative neurons and connectinglines are numbered to simplify illustration. The system control unit 50inputs data to the input layer 1001 of the neural network, and acquiresdata from the output layer 1003.

A number of neurons 1004 in the input layer 1001 is the same as a numberof dimensions of the data to be inputted. The data that is inputted tothe input layer 1001 includes the data of the viewed locus and the locusof the position of the subject. The output layer 1003 outputs 2 values(x coordinate and y coordinate) of the main subject in the video.Therefore a number of neurons 1004 in the output layer 1003 is 2. By theoutput of the 2 values (x coordinate and y coordinate) of the mainsubject, the system control unit 50 can determine the main subject.

A weight w_(ji) is assigned to a connecting line 1005 connecting thei-th neuron 1004 in the input layer 1001 and the j-th neuron 1004 in theintermediate layer 1002. A value z_(j), outputted by the j-th neuron1004 in the intermediate layer 1002, can be calculated by the followingExpression 6.

z _(j) =h(b _(j)+Σ_(i) w _(ji) x _(i))   (Expression 6)

h(p)=max (p, 0)   (Expression 7)

In Expression 6, x_(i) indicates a value that is inputted to the i-thneuron 1004 in the input layer 1001. The i-th neuron 1004 is connectedwith the j-th neuron 1004 in the intermediate layer 1002. If N is anumber of neurons 1004 in the input layer 1001, i is a value in therange of 1 to N. b_(j) is called a “bias”, and is a parameter thatcontrols the ease of firing the j-th neuron 1004.

A function h(p) indicated in Expression 6 and Expression 7 is a functionto output the larger of the value p and the value 0. In other words, thefunction h(p) is a function of which the output value always becomes 0if the input value p in the function is 0 or less, and the output valuebecomes the same value p as the input value if the input value p is morethan 0. The function h(p) is an activation function that is called a“rectified linear unit (ReLU)”. For the activation function, such afunction as a sigmoid function may be used.

The value y_(k) outputted by the k-th neuron 1004 in the output layer1003 can be calculated by the following Expressions 8. Here k inExpression 8 is a value of 1 or 2. k=1 indicates the neuron 1004 whichoutputs a value of the x coordinate of the main subject, and k=2indicates a neuron 1004 which outputs a value of they coordinate of themain subject.

Y_(k) =f(b _(k)Σ_(j) W _(kj) z _(j))   (Expression 8)

In Expression 8, z_(j) indicates a value that is outputted by the j-thneuron 1004 in the intermediate layer 1002. If M is a number of all theneurons 1004 in the intermediate layer 1002 which are connected with thek-th neuron 1004 in the output layer 1003, j is a value in the range of1 to M.

The function f is assumed to be an identity mapping function. Since thecoordinate values of the image are always positive, ReLU used forExpression 7 may be used for the function f Further, in Embodiment 3,only the coordinate values normalized to [0, 1] are handled, asmentioned later, hence a sigmoid function may be used for the functionf.

Here the input data and the correct answer data are provided as learningdata to perform learning in the neural network. For example, viewed lociwhen a person views a plurality of scenes are recorded in advance, andthe viewed loci and the locus of the position of the subject at eachpoint are used as the input data to input to the input layer 1001. Thevolume of the learning data may be artificially increased by a dataaugmentation technique. Each of the recorded loci is normalized to therange of [0, 1] in advance, in accordance with the lateral width and thelongitudinal width of the video (image), so as to eliminate theinfluence of the size of the image. The correct answer data, as thelearning data, indicates the correct answer coordinates (x coordinateand y coordinate) normalized on the image of the main subject. A sceneto be the learning target is preferably a scene in which irregularity ofthe viewed loci is likely to be detected (e.g. scene where a pluralityof subjects cross with each other), but a scene where tracking by theline-of-sight is easy may be included.

When learning is performed, all the weight and bias values are optimizedso as to minimize the loss function L, which indicates the degree ofirregularity between the coordinates outputted based on the input data(locus of viewed position and locus of position of the subject) and thecorrect answer coordinate. For the loss function L, a square sum errorfunction, as indicated in the following Expression 9, can be used.

L(y, t)=½Σ_(k)(y _(k) −t _(k))²   (Expression 9)

In Expression 9, the subscript k indicates a coordinate component, k=1indicates the x coordinate, and k=2 indicates the y coordinate. y_(k)indicates a normalized coordinate value outputted from the neuron 1004in the output layer 1003. t_(k) indicates a coordinate value of thenormalize correct answer of the main subject. By optimizing based onExpression 9, the weights and bias values can be determined so that thecorrect answer coordinates and the outputted coordinate values becomescloser to each other.

The loss function L may be an arbitrary function as long as the functioncan indicate the degree of mismatch (degree of match) between theoutputted coordinate values and the correct answer coordinates.

The learned weights and bias values are stored in the non-volatilememory 56 in advance, and are then stored in the memory 32 as required.Thus by using the learned weights and bias values, the neural networkoutputs the normalized coordinate values (y₁, y₂) of the main subjectbased on Expression 5 to Expression 8.

As described above, according to Embodiment 3, the main subject can bedetermined by using the viewed locus and the locus of the subject. Ifthe main subject can be accurately determined, the digital camera 100can track (capture) the subject intended by the user.

Embodiment 4

When the user frequently moves the position of the digital camera 100,it is likely that the user has lost sight of the main subject. Further,the user is more likely to lose sight of the subject immediately afterpanning or when a telephoto lens is used. Therefore in Embodiment 4, thedigital camera 100 changes whether or not the tracking controlprocessing (processing in S504 to S507) is executed, based on the motioninformation of the digital camera 100 and information on the lens.

<Execution Determination Processing> Processing to determine whether thetracking control processing is performed or not (execution determinationprocessing) will be described with reference to the flow chart in FIG.11. Description on the processing steps the same as Embodiments 1 and 2will be omitted.

In S1101, the system control unit 50 determines whether the motionamount of the digital camera 100, detected by the attitude detectionunit 55, is a threshold THcam or more. In the case where the motionamount of the digital camera 100 is the threshold THcam or more,processing proceeds to S1102. In the case where the motion amount of thedigital camera 100 is less than the threshold THcam, processing proceedsto S1104.

For this motion amount of the digital camera 100, a change in the angleof the digital camera 100 (e.g. pitch angle, yaw angle, roll angle) thatcan be detected by the attitude detection unit 55 can be used. Themotion amount of the digital camera 100 may be a moving amount (shiftamount) of the digital camera 100 in the horizontal direction, thevertical direction, the diagonal direction, and the like. Instead of themotion amount of the digital camera 100, a number of times when themotion amount exceeded a predetermined threshold within a certain timemay be used. This means that as the number of times when the motionamount exceeded the predetermined threshold is higher, the user ismoving the digital camera 100 more frequently. Therefore in this case,in S1101, the system control unit 50 determines whether the number oftimes when the motion amount of the digital camera 100 exceeded thepredetermined threshold, detected by the attitude detection unit 55, isthe threshold THcam or more.

In S1102, the system control unit 50 acquires information on the focaldistance of the lens, from the lens system control circuit 4, anddetermines whether the focal distance is a threshold THd or more. In thecase where the focal distance is the threshold THd or more, processingproceeds to S1103. In the case where the focal distance is less than thethreshold THd, processing proceeds to S1104.

In S1103, the system control unit 50 executes the processing in S504 toS513, including the tracking control processing (S504 to S507) ofEmbodiment 1. In the case where processing proceeds to S1103, it islikely that a telephoto lens is being used because the focal distance islong, and that the motion of the digital camera 100 is large, that is,it is easy for the user to lose sight of the subject. Therefore it ispreferable to execute the tracking control processing based on thedegree of irregularity of the viewed locus. After the processing in S507is executed, processing proceeds to S501 in this flow chart.

In S1104, the system control unit 50 executes the processing in S508 toS513 where the tracking control processing of Embodiment 1 is notincluded.

In Embodiment 4, the tracking control processing in S504 to S507 isexecuted only in the case where the motion amount of the digital camera100 is the threshold THcam or more and the focal distance is thethreshold THd or more. However, the tracking control processing may beexecuted in the case where at least one of the conditions is met: themotion amount of the digital camera 100 is the threshold THcam or more;and the focal distance is the threshold THd or more. Further, just likeEmbodiment 2, the system control unit 50 may change the level of thethreshold THr to determine the irregularity of the viewed locus in S505,in accordance with the motion and focal distance of the camera.

As described above, by changing whether the tracking control processingis executed or not in accordance with the motion of the camera andinformation on the lens, the main subject can be appropriatelydetermined while reducing processing amount.

According to the present invention, a tracking device that is capable ofperforming the tracking control in accordance with the intention of theuser can be provided.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully asanon-transitory computer-readable storage medium') to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-157096, filed on Sep. 18, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A tracking device comprising at least one memoryand at least one processor which function as: an acquisition unitconfigured to acquire a viewed position, which is a position on adisplay viewed by a user; a tracking unit configured to track an objectdisplayed on the display; and a control unit configured to performcontrol processing to control the tracking unit based on a degree ofirregularity of a change of the viewed position.
 2. The tracking deviceaccording to claim 1, wherein in a case where the degree of irregularityof the change of the viewed position is more than a first threshold, thecontrol unit controls so that the tracking unit does not track anobject.
 3. The tracking device according to claim 2, wherein in the casewhere the degree of irregularity of the change of the viewed position ismore than the first threshold, the control unit further controls thedisplay to highlight each object displayed on the display.
 4. Thetracking device according to claim 3, wherein in the case where thedegree of irregularity of the change of the viewed position is more thanthe first threshold, the control unit controls the display to display adisplay item indicating each object displayed on the display,independently of the viewed position.
 5. The tracking device accordingto claim 2, wherein in the case where the degree of irregularity of thechange of the viewed position is more than the first threshold, thecontrol unit further controls so that the user is notified that thechange of the viewed position is irregular.
 6. The tracking deviceaccording to claim 1, wherein in a case where the degree of irregularityof the change of the viewed position is a first threshold or less, thecontrol unit controls the tracking unit to track an object correspondingto the viewed position.
 7. The tracking device according to claim 6,wherein in the case where the degree of irregularity of the change ofthe viewed position is the first threshold or less, the control unitcontrols the display to display a display item, which indicates anobject the tracking unit is tracking.
 8. The tracking device accordingto claim 2, wherein in a case where a degree of irregularity of a changeof a display position of the object is a second threshold or more, thecontrol unit increases the first threshold to be more than a case wherethe degree of irregularity of the change of the display position of theobject is less than the second threshold and is not less than a thirdthreshold, which is smaller than the second threshold.
 9. The trackingdevice according to claim 1, wherein using a machine learning devicewith which machine learning was performed to determine an object thatthe user intends to track with line-of-sight, the control unitdetermines an object that the user intends to track with line-of-sightbased on the change of the viewed position.
 10. The tracking deviceaccording to claim 9, wherein in a case where the tracking unit tracksan object, the control unit controls the tracking unit to track anobject determined using the machine learning device.
 11. The trackingdevice according to claim 1, wherein the degree of irregularity of thechange of the viewed position is a value based on at least one of:frequency of the change of the viewed position, auto correlation of thechange of the viewed position, velocity of the change of the viewedposition and acceleration of the change of the viewed position.
 12. Thetracking device according to claim 1, wherein the degree of irregularityof the change of the viewed position is a value based oncross-correlation between a change of a display position of the objectand the change of the viewed position.
 13. The tracking device accordingto claim 1, wherein the control unit determines whether the controlprocessing is performed or not, based on a degree of irregularity of achange of a display position of the object.
 14. The tracking deviceaccording to claim 13, wherein the control unit does not perform thecontrol processing in a case where the degree of irregularity of thechange of the display position of the object is less than a thirdthreshold.
 15. The tracking device according to claim 1, wherein thecontrol unit determines whether the control processing is performed ornot, based on information on motion of an imaging apparatus that imagesthe object.
 16. The tracking device according to claim 1, wherein thecontrol unit determines whether the control processing is performed ornot, based on information on a focal distance of an imaging apparatusthat images the object.
 17. A tracking method executed by a trackingdevice, which includes a tracking unit that tracks an object displayedon a display, the method comprising: an acquisition step of acquiring aviewed position on the display which is a position viewed by a user; anda control step of performing control processing to control the trackingunit based on a degree of irregularity of a change of the viewedposition.
 18. A non-transitory computer-readable storage medium thatstores programs to execute a tracking method executed by a trackingdevice, which includes a tracking unit that tracks an object displayedon a display, wherein the tracking method includes: an acquisition stepof acquiring a viewed position which is a position on the display viewedby the user; and a control step of performing control processing tocontrol the tracking unit based on a degree of irregularity of a changeof the viewed position.